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Motivation and Goal 


Motivation 


Drum kits may be cool, but they are also extremely costly. Our design 
benefits amateurs who want to experience a cool drum kit but who also 
desire to save money with a cheaper option. We aim to utilize the simplest 
materials such as paper and wood stick to build a useful and cool drum 
system. To use our system, players use a drum stick to hit a piece of paper 
that displays an aerial view of a drum kit. As the player hits the drums on 
the paper, the corresponding drum tones will play. With our design, music 
lovers are able to play the drum for free! 


Goal 


This project aims at creating cost-effective replacement of a real drum kit. 
Musics lovers and everyone else can enjoy this system using affordable 
materials. 


Project Poster 
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b Implementation 
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corresponding drum tones will play. 
With our design, music lovers are 

able to play the drum for free! 


stick hits the 
paper 


microphone 


2. Hit tracking 
only when the 


Virtual Drum Kit System Design 


b Problems and solutions 


Problem Solution 
Iteration delay in * Always track and save the stick 
stick position position 


tracking algorithm * Do template matching with 
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Place a microphone inside the box 
to receive and detect the sound of 
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b Testing and Future Work 


We tested our system in a quiet study room (with little surrounding 
noise). Our virtual drum kit system works well with only minimal 
audio and video processing delay. 

In the future, we will consider: 


Delay: All implementations are based on MATLAB, which is not 
suitable for real time processing. We may move our 
implementations to LABVIEW to achieve better results in real 
time. 

Multi-stick: We may include another stick in our system to 
allow players to use two sticks simultaneously. 


We divided our system design into three major components: video processing, which tracks the current position of drum stick in real time; audio 
processing, which detects the ‘hit’ of the drum stick and generates drum sounds; and an integration of the previous 2 parts. 


1) Video: For each snapshot taken from the video, the 

processing returns the coordinates of the drum stick 

position relative to the paper boundary. 

° Position calibration using canny edge detection 

. Template (drum stick head) matching to detect the 
position of drum stick 

2) Audio: Each time the stick hits the paper, the sound 

impulse created by the collision is detected by the 

microphone. The sound impulse is used in conjunction 

with the coordinates of the drum stick position to 

generate the appropriate drum sound. 

‘ From this figure, 

We could see that 

the collision is 

detected by the 

microphone 


3) Integration: A webcam will videotape the paper and 
stick movement; then, a computer will process the 
images to determine the drum stick positions. Once the 
microphone detects the audio signal created by the 
player hitting the drums on the paper, the visual 
information showing the current position of the drum 
stick is used to generate the correct sounds. 
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System Setup 


Major Components 
In general, we divided our system design into three major components: 


e Video processing, which tracks the current position of drum stick in 
real time; 

e Audio processing, which detects the ‘hit’ of the drum stick and 
generates drum sounds; 

e Integration of the previous 2 parts. 


Required Materials 
There are some required materials for our system setup: 


e A webcam: a camera is used to videotape the paper and stick 
movements. In our experiment we simply used the webcam embedded 
in our laptop. However, other users may use any other types of 
cameras depending on individual requirements. 

e A microphone: since the camera can only track the stick's position but 
not being able to know whether the stick really hits the paper to make 
sound, a microphone is used to detect when the stick actually hits the 
paper (when the player wants to play a note). We placed the 
microphone inside a sound-prove paper box to avoid noise 
interference. 


Virtual Drum Kit System Design 
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webcam 


2. Hit tracking 
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; 4. Generate sound 
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from each hit to current stick position 
generate sound on the drum 


Notice how different parts are connected with each other. 


Note:Our whole system works under the sequence numbered(from 1 to 4) 
in the above illustration. 


Implementation: Detect Position of Drum Stick 


Image Processing--Calibration and Canny Edge Detection 


For each snapshot taken from the video, the processing returns the 
coordinates of the drum stick position relative to the paper boundary. 


Since we are only using a part of the screen (only the part of white paper). 
We need to calibrate a new coordinate system for our later position 
detection. Therefore, we first took a snapshot of the whole system to detect 
the edge of the paper and then chose a corner as the new origin. 


Intro to Canny Edge Detection 


To detect the edge of the paper, we used canny edge detector. To better 
understand the method, instead of using the MATLAB command, we 
implemented the method on our own. 


Basic Steps for Canny Edge Detector 


e Smooth with Gaussian Filter 

e¢ Compute Horizontal/Vertical Gradient 
e Compute magnitude of gradient 

e¢ Non-Maximal Suppression 

e Perform Hysteresis Threshold 


Note:We have our codes attached at the bottom. 


The image we implemented Canny Edge Detection 


After Canny Edge Detection 


The edge of the pater has been detected fairly accurately. 


Boforehand 

Before doing any image processing techniques on an image, we usually first 
smooth the image to get rid of most noise. To do this, we apply a Gaussian 
smoothing filter to the image. 


Image Gradient 

An image gradient is a directional change in the intensity or color in an 
image. Edge detection utilized image gradient a lot since edges usually are 
places with discontinuity. 

Gradient calculations 


% The filter that reduces noise 
Ff = 1/159*[2 45 42;491294;5 12 1512 55491294:'2 454 
2]; 


n = imfilter(mg,f,'same'); 


%Gradient Magnitude and Angle 

%x direction partial derivative 

mg = double(mg); 

Dx = [-1 0 1;-2 0 2;-1 0 1]; 

% y direction partial derivative 

Dy = [1 2 1;0 0 0;-1 -2 -4]; 

% Compute the gradient magnitude and the angle of the gradient 
ni = imfilter(mg, Dx); 

n2 = imfilter(mg,Dy); 

% initialization 


s= (1; 
theta = []; 
theta1= []; 


ni = double(n1); 
n2 = double(n2); 


for 1 = 1:size(m,1) 
for j = 1:size(m,1) 
% calculate the gradient magnitude 
S(i,j) = sqrt((n1(i,j))%2+(n2(1,j))%2); 
theta(i,j) = 360*(atan(n1i(i,j)/n2(i,j)))/3.14; 
% round angles 


if (22.5&1t;theta(i, j)&amp;&amp; theta(i, j )&1t;=67.5) 
theta1(i,j) = 45; 
elseif (67.5<theta(i, j)&&theta(i, j )<=112.5) 
theta1(i,j) = 90; 
elseif (112.5<theta(i, j)&&theta(i, j)<=157.5) 
thetai(i,j) = 135; 
elseif (157.5<theta(i, j)&&theta(i, j)<=202.5) | | 
(O<theta(i,j)&&... 
theta(i, j)<=22.5) | | 
(337.5<theta(i, j)&&theta(i, j )<=360) 
theta1(i,j) = 0; 
elseif (202.5<theta(i, j)&&theta(i, j )<=247.5) 
theta1(i,j) = 45; 
elseif (247.5<theta(i, j)&&theta(i, j)<=292.5) 
theta1(i,j) = 90; 
else 
theta1(i,j) = 135; 
end 
end 
end 


Non-Maximum Suppression 


This method returns a single pixel width edge of the image. Basic algorithm 
is as follows: If pixel (x; y) has the highest gradient magnitude of the three 
pixels examined, it is kept as an edge. If one of the other two pixels has a 
higher gradient magnitude,then pixel (x; y) is not on the center of the edge 
and should not be classified as an edge pixel. In this way we could get a 
clear and thin edge of the image. 


We round the angle to 0, 45, 90 and 135. With a specific angle value, we 
examine the maxima within the range of three pixel values in that direction 
(vertically, horizontally or diagonally). By picking up the maxima and 
discarding non maxima points, the edge will be one pixel wide. In order to 
enhance the detection, we use hysteresis thresholding. 

Non-Maximum Suppression 


S = padarray(s,[1 1]); 
theta1 = padarray(theta1,[1 1]); 


for k = 2:(size(m,1)+1) 
for 1 = 2:(size(m,1)+1) 


if ((theta1(k,1) == 90) && (s(k,1)== 
max(max(s(k+1,1),s(k,1)),... 
s(k-1,1)))) 
mge(k,1) = s(k,1); 
elseif ((thetai(k,1) == 0) && (s(k,1)== 
max(max(s(k,1+1),... 
s(k,1)),s(k,1-1)))) 
mge(k,1) = s(k,1); 
elseif ((thetai(k,1) == 135) && (s(k,1) == 
max(max(s(k+1,1+1),... 
s(k,1)),s(k-1,1-1)))) 
mge(k,1) = s(k,1); 
elseif ((theta1(k,1) == 45) && (s(k,1) == 
max(max(s(k+1,1-1),... 
s(k,1)),s(k-1,1+1)))) 
mge(k,1) = s(k,1); 
end 
end 
end 


Note:** When writing the MATLAB program for this part, finding the 
maxima at the boundary rows and columns require padding to the image 
matrix. So we padded zeros to the matrix to ensure that the boundary edges 
could also be detected. 


Hysteresis Thresholding 

Here we set two threshold values, a low threshold and a high threshold. The 
threshold values are important for the resulting image since if the low value 
is too low there could be invalid edges not being discarded, and if the high 
value is too high the edge will be discontinuous and some valid edges could 
be mistakenly discarded. Moderate threshold values are required. 
Hysteresis Thresholding 


% Hysteresis Thresholding 
% Hysteresis is one way of solving this problem. Instead of 


choosing a single 
% threshold, two thresholds thigh and tlow are used. 


high = 105; 
low = 35; 
tmg = mge; 


% find the edge to be discarted 


indexi = find(mge<=low); 
tmg(index1) = 0; 
for 1 = 3:(size(m,1)-2) 
for j = 3:(size(m,1)-2) 
% neighbors in a 3*3 region around it have gradient 
magnitudes 
% greater than threshold high ,keep the edge 
if (mge(i,j)<high && mge(i, j)>Llow) 
if ((mge(i-1:1+1, j-1:j+1)>high) == zeros(3) ) 
tmg(i,j)=0; 
% If non of pixel (x; y)'s neighbors have high 
gradient 
% magnitude but at least one falls between low 
and high, 
% search the 5*5 region to see if any of these 
pixels have 
% a magnitude greater than high. If so, keep 
the edge. 
elseif ((mge(i-1:1+1,j-1:j+1)>low) ~= [0 0 0;0 1 


0;0 0 O]) 

if ((mge(i-2:1+2,j-2:j+2)>high) == zeros(5)) 
tmg(i,j)=0; 
end 
end 
end 
end 
end 


Note:It will be better if the webcam is placed in parallel with the paper; in 
this way the resulting edge detected will be a desired regular rectangular. 


Image Processing--template matching 


After calibration, we are set up in the new coordinate system, which is the 
white paper region. We will detect drum stick position in this region. 


To detect the position of drum stick, we used template matching. Since the 
stick head is black that has a high contrast from its surroundings (paper is 
white), we build a simple square template filled with black pixels. This 
template serves as a filter that helps us to find the position of stick head. We 
did 2D convolution between this template filter and the snapshot from the 
video. Euclidean distance is used to find the matching part. Below is a how 
we did template matching step by step: 


Matching 

As the template moves along x/y axis of the image, assume that it has size 
of N by N; the template will compare all its pixel values with N by N 
matrix of the image it moves to. 

Basic algorithm steps 


Set the minimum distance to infinity 

Iterate the template through the image 

Each iteration calculate for the euclidean distance between the template and 
the part of matrix the it iterates to 

If the distance is smaller than the current minimum distance, replace the 


minimum distance with that value 
Repeat the iteration until the 2D convolution is finished 


Template matching Code 


1=0; 


while(i < 100) 


end 


% trigger fun: start logging now 
trigger (vid) 
frame = getsnapshot(vid); % record the current frame 
flushdata(vid); 
delete(vid); 
Im = double(frame); 
Im = Im/max(max(Im)); % normalize the image 
YCCLUUULUUUUUUHUNN%%P FOCeSSiIN g TM%%KURUHUMUMG V0 VO10% 
for m = 1:size(Im,1)-N+1 
for n = 1:size(Im,2)-N+1 
diff = sum(sum(Im(m:m+N-1,n:n+N-1)-temp) ); 
if diff<best_error 
loc_x = m+N/2; 
loc_y = n+N/2; 


end 
end 
end 
loc_y = loc_y-calib(1); 
loc_x = loc_x-calib(2); 


position(i,1) = loc_x; 

position(i,2) = loc_y; 

LULU CUUUUUUMUUUHKYHENAUProcessin 9%%%%%H2%%67670%7070707070070% 
imshow(Im); 

hold on 

rectangle('Position', [size(Im,1)-loc_x,loc_y,N,N]) 
1=i+1; 


Implementation: Detecting a Hit 


Audio Processing 


Each time the stick hits the paper, the sound impulse created by the 
collision is detected by the microphone. The sound impulse is used in 
conjunction with the coordinates of the drum stick position to generate the 
appropriate drum sound. We used template matching by calculating the 
Euclidean distance. 


Ambient Noise Level 

Also called background noise level, is utilized to find the relative trigger 
level when a peak reached a certain height. We first test the ambient noise 
in the environment by recording a period of sound and take an average. The 
noise level will determine the sound trigger level. The larger the noise, the 
harder the player may need to hit the drum to produce a sound. 
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The big spike contained in the red box indicates the stick 
hit the drum kit. 


Audio Input and Detecting Hit 


% config and caliberate audio input 
% return ai audio object and th: threshhold volume calculated 
from 
% Caliberation 
function [ai,th] = configAudio(samplePeriodConstant ) 
ai = analoginput('winsound'); % create an analog 
input device 
addchannel(ai,1); 
Fs=8000; 
set(ai, 'TriggerType', 'manual'); 
set(ai, 'SamplesPerTrigger', 8000); 
set(ai, 'TriggerDelay',0); 
set(ai, 'TriggerDelayUnits', 'samples'); 
ai.TriggerRepeat = inf; 
start(ai); 
% wait ready 
pause(1); 
% trigger data 
trigger(al); 
% get recent 1x8000 sample 
data = getdata(ai); 
% caliberated volume 
th = mean(abs(data)); 
% new trigger length for constantly checking impulses 
stop(ai); 
set(ai, 'SamplesPerTrigger', samplePeriodConstant ); 
start(al); 
pause(1); 
end 


Implementation: Integration 


Connection between audio and video 


A webcam will videotape the paper and stick movement; then, a computer 
will process the images to determine the drum stick positions. Once the 
microphone detects the audio signal created by the player hitting the drums 
on the paper, the visual information showing the current position of the 
drum stick is used to generate the correct sounds. 


Reduce processing delay 


Whether the stick hits the paper or not, we processed the snapshots and 
saved the current stick location in an array. Only when the stick hits the 
paper, which is detected by the impulse from the microphone, we took out 
the position information to generate the correct sounds. 


Sound division 


Below is how we divide our paper. Each part corresponds to a unique 
sound. In total there are four parts. The stick position determines which 
sound to play by the computer. 


Playing different sounds 


1=0; 
playing=0; 
sample_rate_devider=3; 
while(1==1) 
1 > I+; 
if (sample_rate_devider==3) 
hitPosition = tracking(vidobj,edgePosition) ; 
sample_rate_devider=0; 
end 


sample_rate_devider=sample_rate_devider+1; 


pastData = sampleAudio(pastData, ai, dataLengthConstant); 
% if we got a audio trigger 
if(playing ~=0) 
playing=playing-1; 
else 
if(testImpulse(pastData, threshHold)==1) 
playing = 5; 
soundType = 
getwhichSound(hitPosition, edgePosition); 


% makeSound(soundType) ; 
if (soundType==1 ) 
%play sound1 
sound(sound1,Fs1); 
elseif (soundType==2 ) 
%play sound 2 
sound(sound2,Fs2)j; 
elseif (soundType==3) 
%play sound 3 
sound(sound3,Fs3)j; 
else (soundType==4 ) 
%play sound 4 
sound(sound4,Fs4); 
end 


end 
end 
end 


Testing, Conclusion and Future Work 


Testing and Conclusion 


We tested our system in a quiet study room (with little surrounding noise). 
Our virtual drum kit system works well with only minimal audio processing 
delay. During our demonstration, we tested our system in Duncan Hall 
where we expected the noise and echo to severely affect our system. 
However, the result appeared to be fairly good with acceptable audio delay 
and ‘hit' detect error. 


In the future, we will consider: 


Based on our current testing results, we saw potential enhancement that we 
may keep working on the system in the future. 


e Delay: All implementations are based on MATLAB, which is not 
suitable for real time processing. We may move our implementations 
to LABVIEW or Cpp to achieve better results in real time. 

e Multi-stick: We may include another stick in our system to allow 
players to use two sticks simultaneously, which better simulates a real 
drum kit. 
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